batch-cluster
Advanced tools
Support external batch-mode tools within Node.js.
Many command line tools, like ExifTool and GraphicsMagick, support running in a "batch mode" that accept commands provided through stdin and results through stdout. As these tools can be fairly large, spinning them up can be expensive (especially on Windows).
This module expedites these commands, or "Tasks," by managing a cluster of these "batch" processes, feeding tasks to idle processes, retrying tasks when the tool crashes, and preventing memory leaks by restarting tasks after performing a given number of tasks or after a given set of time has elapsed.
This package powers exiftool-vendored, whose source you can examine as an example consumer.
Depending on your yarn/npm preference:
$ yarn add batch-cluster
# or
$ npm install --save batch-cluster
The child process must use stdin and stdout for control/response.
BatchCluster will ensure a given process is only given one task at a time.
If these links are broken, use https://batch-cluster.js.org/
Create a singleton instance of BatchCluster.
Note the constructor options takes a union type of
The default logger writes warning and
error messages to console.warn and console.error. You can change this to
your logger by using setLogger.
Implement the Parser class to parse results from your child process.
Construct a Task with the desired command and the parser you built in the previous step, and submit it to your BatchCluster singleton's enqueueTask method.
See src/test.ts for an example child process. Note that the script is designed to be flaky on order to test BatchCluster's retry and error handling code.
MAJOR or API version is incremented forMINOR or UPDATE version is incremented forPATCH version is incremented for/PID option seemed to work downcased, but the docs say
to use uppercase, so I've updated it.(v2.1.2 is the same contents, but np had a crashbug during publish)
end for BatchProcess, which may prevent very long-lived
consumers from sporadically leaking child processes on Mac and linux.Logger.trace and moved logging related to per-task items down
to trace, as heavy load and large request or response payloads could
overwhelm loggers. If you really want to see on-the-wire requests and results,
enable trace in your debugger implementation. By default, the
ConsoleLogger omits log messages with this level.BatchClusterObserver with a simple EventEmitter API on
BatchCluster to be more idiomatic with node's APImaxTaskErrorsPerProcess parameter was removed)Rate is simpler and more accurate now.BatchClusterObserver for error and lifecycle monitoringtimers.setInterval. May address this
issue.
Thanks for the PR, Tim Fish!BatchProcess.end() to use until() rather than Promise.race,
and always use kill(pid, forced) after waiting the shutdown grace period
to prevent child process leaks.Logger.setLogger() for debug, info, warning, and errors. debug and
info defaults to Node's
debuglog,
warn and error default to console.warn and console.error,
respectively.tsc provides good lint coverage nowdelay now allows
unrefing the
timer, which, in certain circumstances, could prevent node processes from
exiting gracefully until their timeouts expiredkill() and running() from BatchProcesstaskkill on windows and kill -9
on other unix-like platforms if they don't terminate after sending the
exitCommand, closing stdin, and sending the proc a SIGTERM. Added a test
harness to exercise.mocha tests don't require the --exit hack anymore 🎉.running() works correctly for PIDs with different owners now.yarn upgrade --latestprettier and delintedconsole.log with a call to log.maxProcs wasn't always utilized by onIdle, which meant in
certain circumstances, only 1 child process would be servicing pending
requests. Added breaking tests and fixed impl.kill(0) calls to verify the child
processes are still running work across different node version and OSesBatchProcess (whose API should not be accessed
directly by consumers, so the major version remains at 1).end()BatchCluster to BatchProcessBatchCluster now has a force-shutdown exit handler to accompany the
graceful-shutdown beforeExit handler. For reference, from the
Node docs:The 'beforeExit' event is not emitted for conditions causing explicit termination, such as calling process.exit() or uncaught exceptions.
Rate's time decay in the interests of simplicityprocessFactory or versionCommand fails more often than a given
rate, BatchCluster will shut down and raise exceptions to subsequent
enqueueTask callers, rather than try forever to spin up processes that are
most likely misconfigured.maxProcAgeMillis, and restarted as neededBatchCluster now practices good listener hygene for process.beforeExitv2.2.0
/PID option seemed to work downcased, but the docs say
to use uppercase, so I've updated it.(v2.1.2 is the same contents, but np had a crashbug during publish)
FAQs
Manage a cluster of child processes
The npm package batch-cluster receives a total of 0 weekly downloads. As such, batch-cluster popularity was classified as not popular.
We found that batch-cluster demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.
Security News
Michigan TypeScript founder Dimitri Mitropoulos implements WebAssembly runtime in TypeScript types, enabling Doom to run after processing 177 terabytes of type definitions.
Security News
vlt's new "reproduce" tool verifies npm packages against their source code, outperforming traditional provenance adoption in the JavaScript ecosystem.